How to use this slides

• Press Esc to navigation mode
• Press pageDown, Right or Down to go to next slide
• Presss PageUP, Left or Up to go to previous slide

Outline

• Recap for observation skills (10 min)
• R Basic (quick recap) and RStudio IDE (10 min)
• Connection (20 min)
  • Crawler’s toolkits in R
• Parsing Data (50 min)
  • XPath & CSS
  • json
  • xml
  • html table
  • regex
• Save your data (10 min)
• Case study
• References

Install: Chrome Extension

• EditThisCookie
• XPath Helper
• JSON Viewer
• Advanced REST client

Recap Observation Skills and HTTP request for Connection

• URL, Method, GET, POST, Headers, Body
• Excample: PTT Gossiping

R Packages Required

Pipeline Coding

• magrittr

Crawler’s toolkits in R

• rvest: a web scraper based on httr and xml2
• httr: toolkit of HTTP methods in R
• XML & : XML parser
• xml2: xml parser based on libxml2

misc

• stringr: string manipulaiton
• data.table: extension of data.frame, a powerful ETL tool in R

Install Packages

(if you haven’t installed those packages yet)

## === install required packages ===
pkg_list <- c("magrittr", "httr", "rvest", "stringr", "data.table",
              "jsonlite", "RSQLite")
pkg_new <- pkg_list[!(pkg_list %in% installed.packages()[,"Package"])]
if(length(pkg_new)) install.packages(pkg_new)
rm(pkg_new, pkg_list)

Let’s Rock with R!

Hello RStudio

RStudio Settings

Must-known keyboard shortcuts

All RStudio keyboard shortcuts

R recap

• How to get help
• Working Environment
• Basic Data Structure
• Function
• if/else, for, tryCatch
• magrittr
• data.table

How to get help

• ?: Access ducument in R console
• ??: Search the halp system
• google with appropriate keyword. For example:
  • R {package name}
  • R {algorithm name}
• Forum
  • Stack Overflow
    A question and answer site for programmers
  • Taiwan R user Group
    A free R/data analysis user group in Taiwan
  • PTT R_Language board
    A bbs forum for R in Taiwan

Working Environment

Check your working directory everytime you start to work!

Using getwd/setwd to get/set your working directory.

wd = getwd()
setwd("resources/img")
getwd()
setwd(wd)
getwd()

RStudio will set working directory automatically when opening new files.

If you using Projects, RStudio will change working directory for you automatically.

Basic Data Structure

Vector, Matrix, Array, List and Data frame are the most basic data structure in R. These data structures can be mapped into a table according to:

• dimension
• Homogeneous or Heterogeneous of data type

(Atomic) Vector

• (Atomic) vector is a basic data structure in R.
• They are linear vectors of a single primitive type. A scalar is a vector of length 1.

v1 <- c(1:10)
v1
#>  [1]  1  2  3  4  5  6  7  8  9 10
is.vector(v1)
#> [1] TRUE
length(v1)
#> [1] 10
s1 <- 2
s1
#> [1] 2
is.vector(s1)
#> [1] TRUE
length(s1)
#> [1] 1

List

Lists are also vectors, but not atomic vectors. Lists are generic vectors, with (naturally) different semantics.

Elements in a list can be any kinds type and its length is arbitrary.

Function str can help you investigate the structure of a nested list.

li <- list(a = 1:10, 
           b = c("apple", "banana"))
str(li)

#> List of 2
#>  $ a: int [1:10] 1 2 3 4 5 6 7 8 9 10
#>  $ b: chr [1:2] "apple" "banana"

li2 <- list(li = li, 
            c = matrix(1:4, nrow = 2))
str(li2)

#> List of 2
#>  $ li:List of 2
#>   ..$ a: int [1:10] 1 2 3 4 5 6 7 8 9 10
#>   ..$ b: chr [1:2] "apple" "banana"
#>  $ c : int [1:2, 1:2] 1 2 3 4

Visualising lists

x1 <- list(c(1, 2), c(3, 4))
x2 <- list(list(1, 2), list(3, 4))
x3 <- list(1, list(2, list(3)))

Lists Subsetting

a <- list(a = 1:3, b = "a string", c = pi, d = list(-1, -5))

• [ extracts a sub-list. The result will always be a list.

str(a[1:2])
str(a[4])

• [[ extracts a single component from a list. It removes a level of hierarchy from the list.

y <- list("a", 1L, 1.5, TRUE)
str(y[[1]])
str(y[[4]])

• $ is a shorthand for extracting named elements of a list. It works similarly to [[ except that you don’t need to use quotes.

a$a
a[["b"]]

Lists Subsetting (Pepper Shaker)

Data Frame

Data frame is a 2-dimension data structure to deal with a table-like heterogeneous data.

df <- data.frame(gender = c("male", "female", "female", "male"),
                 age = c(33, 18, 24, 26))
## Add new column in a data frame
df$city <- c("Taipei", "Taipei", "Hsinchu", "Taichung")

df

#>   gender age     city
#> 1   male  33   Taipei
#> 2 female  18   Taipei
#> 3 female  24  Hsinchu
#> 4   male  26 Taichung

str(df)

#> 'data.frame':    4 obs. of  3 variables:
#>  $ gender: Factor w/ 2 levels "female","male": 2 1 1 2
#>  $ age   : num  33 18 24 26
#>  $ city  : chr  "Taipei" "Taipei" "Hsinchu" "Taichung"

Recap: data structure

All data structures above are objects. They apply different methods and saved as different type internally.

Function

To understand computations in R, two slogans are helpful:

• Everything that exists is an object.
  
• Everything that happens is a function call.

John Chambers

`+`
#> function (e1, e2)  .Primitive("+")
`<-`
#> .Primitive("<-")
`[`
#> .Primitive("[")
`c`
#> function (..., recursive = FALSE)  .Primitive("c")

Function in R

A typical function in R may look like:

f <- function(par1, par2, ...) {
    # Some magic happened
    return(sth)    # return something
}

• If you don’t use return to specify the return value, the return value will be the last expression inside the function.
• Call by value by default
  send a copy of a object into a function and the input object won’t change after execution without assignment

If

The basc structure of conditional execution in R is:

if (an expression returns TRUE or FALSE) {
    # do something
} else if (another expression returns TRUE or FALSE) {
    # do something
} else {
    # do something
}

for

Iterate items in R.

# iterate a character vector
for (i in c("a", "b")) {
    print(i)
}

#> [1] "a"
#> [1] "b"

# nested loop
m <- matrix(numeric(), nrow = 2, ncol = 2)
for (i in 1:nrow(m)) {
    for (j in 1:ncol(m)) {
        m[i, j] <- i * j
    }
}
m

#>      [,1] [,2]
#> [1,]    1    2
#> [2,]    2    4

tryCatch

• tryCatch let you deal with error handling in R.
• It is IMPORTANT when wrapping your crawler into a function because you don’t want your function fail without any outcome.

tryCatch({
  result <- expr
  # If you want to use more than one 
  # R expression in the "try" part then you'll have to 
  # use curly brackets.
  # 'tryCatch()' will return the last evaluated expression 
  # in case the "try" part was completed successfully
}, warning = function(w) {
  message("Here's the original warning message:")
  message(w)
  # Choose a return value in case of warning
  return(NULL)
}, error = function(e) {
  message("Here's the original error message:")
  message(e)
  # Choose a return value in case of error
  return(NA)
},
}, finally {
  message("Some other message at the end")
  # finally:
  # Here goes everything that should be executed at the end,
  # regardless of success or error.
  # If you want more than one expression to be executed, then you 
  # need to wrap them in curly brackets ({...}); otherwise you could
  # just have written 'finally=<expression>' 
})

https://stackoverflow.com/questions/12193779/how-to-write-trycatch-in-r/12195574#12195574

magrittr

What is magrittr

Pipe argument to right-hand side with %>%

• x %>% f is equivalent to f(x)
• x %>% f(y) is equivalent to f(x, y)
• x %>% f %>% g %>% h is equivalent to h(g(f(x)))
• x %>% f(y, .) is equivalent to f(y, x)
• x %>% f(y, z = .) is equivalent to f(y, z = x)

Example

library(magrittr)
iris %>% head()

• Using the dot place-holder

"Ceci n'est pas une pipe" %>% gsub("une", "un", .)

Workflow of Crawler Design

Architecture of Crawlers

Workflow

1. 找到資料頁，想像資料要長什麼樣子，設想產出的資料格式(schema)
2. 觀察網頁內容，找到資料所在的request/response，再一層層往上解析，套上判斷式及迴圈, 完成爬蟲的自動化。
3. 解析取得的資料。

Crawler’s toolkits in R

• rvest: a web scraper based on httr and xml2
• httr: toolkit of HTTP methods in R
• XML & : XML parser
• xml2: xml parser based on libxml2
• RCurl: a wrapper of libcurl

HTTP request

A valid HTTP request includes four things:

• URL: a unique address for a thing
• Method:
  • GET: to retrieve a resource
  • POST: to create a new resource
• Headers: Meta-information about a request
  • User-Agent
  • Cookie
  • Content-type
  • List of HTTP header fields
• Body: data to be send to a server

Web Connector in R

• {httr}
• {RCurl}
• download.file()
• Socket Connection Tools:
  • socketConnection
  • make.socket / read.socket / close.socket

Connection: GET Method

Use GET() to request data from a specific resource

起手式

## Not Run
library(httr)
library(rvest)
res <- GET(
  url = "http://httpbin.org/get",
  add_headers(a = 1, b = 2),
  set_cookies(c = 1, d = 2),
  query = list(q="hihi")
)
content(res, as = "text", encoding = "UTF-8")
content(res, as = "parsed", encoding = "UTF-8")

一個例子學會第一隻爬蟲

• PTT Gossiping

library(magrittr)
library(httr)
library(rvest)

#> Loading required package: xml2

url <- "https://www.ptt.cc/bbs/Gossiping/index.html"
doc <- GET(url, 
           set_cookies('over18'='1')) %>%  # over18 cookie
  content(as = "text", encoding = "UTF-8") %>% 
  read_html

doc %>% 
    html_nodes(xpath = '//*[@id="action-bar-container"]/div/div[2]/a[2]') %>% 
    html_attr("href")

#> [1] "/bbs/Gossiping/index14934.html"

httr

• Basic Features:
  • HTTP verbs: GET(), POST(), HEAD()…
  • http_status: Translate http status code HTTP status code
  • set_cookies(): set cookles
  • add_headers(): add additional headers to a request.
  • headers(): Access response headers
  • content(): Retrieve the contents of a request
    • as = “parsed”: detect content type automatically and parse the result
    • as = “text”: return character
    • as = “raw”: return binary

Parsing Response Content

httr basics

library(httr)
res <- GET(
  url = "http://httpbin.org/get",
  add_headers(a = 1, b = 2),
  set_cookies(c = 1, d = 2),
  query = list(q="hihi")
)

status_code(res)

headers(res)

content(res, type = "text", encoding = "UTF-8")

Set header

Sometime you may need to provide appropriate HTTP header fields with add_headers() to make a request.

• Starbucks

Cookies

• A small piece of data sent from a website and stored in the user’s web browser while the user is browsing it
• Someetimes you may need set cookies with set_cookies to retreve a request.

Connection: POST method

• Submits data to be processed to a specified resource
• The query string (name/value pairs) is sent in the HTTP message body

起手式

## Not Run
library(httr)
library(rvest)


res1 <- POST(url = "http://httpbin.org/post", 
              add_headers(a = 1, b = 2),
              set_cookies(c = 1, d = 2),
              body = "A simple text string")

res2 <- POST(url = "http://httpbin.org/post",
             add_headers(a = 1, b = 2),
             set_cookies(c = 1, d = 2),
             body = list(x = "A simple text string", 
                         y = "hihi"))

identical(res1, res2)
content(res2, as = "text", encoding = "UTF-8")
content(res2, as = "parsed", encoding = "UTF-8")

一個例子學會第二隻爬蟲: Guestbook

Guestbook

Exercise: Guestbook

Try to post a message in App Engine GuestBook

The response status code

res <- GET("http://httpbin.org/get")

# Get an informative description:
http_status(res)
#> $category
#> [1] "Success"
#> 
#> $reason
#> [1] "OK"
#> 
#> $message
#> [1] "Success: (200) OK"

# Or just access the raw code:
res$status_code
#> [1] 200

# highly recommend using one of these functions whenever you're using httr inside a function to make sure you find out about errors as soon as possible.
warn_for_status(res)
stop_for_status(res)

The response Body

3 ways to access the body of the request:

• text

res <- GET("http://httpbin.org/get")
content(res, "text", encoding = "UTF-8")  # accesses the body as a character vector
#> [1] "{\n  \"args\": {}, \n  \"headers\": {\n    \"Accept\": \"application/json, text/xml, application/xml, */*\", \n    \"Accept-Encoding\": \"gzip, deflate\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"libcurl/7.43.0 r-curl/0.9.7 httr/1.1.0\"\n  }, \n  \"origin\": \"123.193.211.51\", \n  \"url\": \"http://httpbin.org/get\"\n}\n"

• binary

(bin <- content(res, "raw"))  # raw (binary) vector
#>   [1] 7b 0a 20 20 22 61 72 67 73 22 3a 20 7b 7d 2c 20 0a 20 20 22 68 65 61
#>  [24] 64 65 72 73 22 3a 20 7b 0a 20 20 20 20 22 41 63 63 65 70 74 22 3a 20
#>  [47] 22 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 2c 20 74 65 78 74
#>  [70] 2f 78 6d 6c 2c 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d 6c 2c 20
#>  [93] 2a 2f 2a 22 2c 20 0a 20 20 20 20 22 41 63 63 65 70 74 2d 45 6e 63 6f
#> [116] 64 69 6e 67 22 3a 20 22 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 22 2c
#> [139] 20 0a 20 20 20 20 22 48 6f 73 74 22 3a 20 22 68 74 74 70 62 69 6e 2e
#> [162] 6f 72 67 22 2c 20 0a 20 20 20 20 22 55 73 65 72 2d 41 67 65 6e 74 22
#> [185] 3a 20 22 6c 69 62 63 75 72 6c 2f 37 2e 34 33 2e 30 20 72 2d 63 75 72
#> [208] 6c 2f 30 2e 39 2e 37 20 68 74 74 72 2f 31 2e 31 2e 30 22 0a 20 20 7d
#> [231] 2c 20 0a 20 20 22 6f 72 69 67 69 6e 22 3a 20 22 31 32 33 2e 31 39 33
#> [254] 2e 32 31 31 2e 35 31 22 2c 20 0a 20 20 22 75 72 6c 22 3a 20 22 68 74
#> [277] 74 70 3a 2f 2f 68 74 74 70 62 69 6e 2e 6f 72 67 2f 67 65 74 22 0a 7d
#> [300] 0a
# writeBin(bin, "myfile.txt") # the highest fidelity way of saving files to disk

• auto parsed httr provides a number of default parsers for common file types:
  html_document, csv, json, jpeg, png. See ?httr::content

content(res, "parsed")
#> $args
#> named list()
#> 
#> $headers
#> $headers$Accept
#> [1] "application/json, text/xml, application/xml, */*"
#> 
#> $headers$`Accept-Encoding`
#> [1] "gzip, deflate"
#> 
#> $headers$Host
#> [1] "httpbin.org"
#> 
#> $headers$`User-Agent`
#> [1] "libcurl/7.43.0 r-curl/0.9.7 httr/1.1.0"
#> 
#> 
#> $origin
#> [1] "123.193.211.51"
#> 
#> $url
#> [1] "http://httpbin.org/get"

The secret of URL

URL?par1=val1&par2=val2

• The query string is sent in the URL of a GET request
• Non-ASCII character and some preserved characters need to be encoded with URL Encode
• In most situation, you may need to manipulate the query string when you write a crawler

Query String

You can assign query parameter with query()

res1 <- GET(
  "http://ecshweb.pchome.com.tw/search/v3.3/all/results?q=sony&page=1&sort=rnk/dc"
)

url2 <- "http://ecshweb.pchome.com.tw/search/v3.3/all/results"
res2 <- GET(url2,
           query = list(q="sony", page="1", sort="rnk/dc"))
identical(res1$content, res2$content)

#> [1] TRUE

Url Encoding in R

URLencode(" ")

#> [1] "%20"

greeting = "你好嗎我很好"
greeting_enc = URLencode(greeting)
greeting_enc

#> [1] "%E4%BD%A0%E5%A5%BD%E5%97%8E%E6%88%91%E5%BE%88%E5%A5%BD"

URLdecode(greeting_enc)

#> [1] "你好嗎我很好"

Concatenate strings / String formatting

paste0("hihi", greeting)

#> [1] "hihi你好嗎我很好"

paste0("hihi", greeting, 1:3)

#> [1] "hihi你好嗎我很好1" "hihi你好嗎我很好2" "hihi你好嗎我很好3"

paste("hihi", greeting, 1:3, sep = " ", collapse = ",")

#> [1] "hihi 你好嗎我很好 1,hihi 你好嗎我很好 2,hihi 你好嗎我很好 3"

sprintf("%s,%s嗎?", "hihi", greeting)

#> [1] "hihi,你好嗎我很好嗎?"

Parsing Response Content

What to Parse

結構化資料

• html elements: (rvest::html_node)
• json, jsonp: PChome, 紫外線即時監測資料 (jsonlite::fromJSON)
• html table: 公開資訊觀測站 (rvest::html_table, XML::readHtmlTable)
• xml table: 7-11 (XML::xmlToDataFrame)

非結構化資料

• RegEx: 7-11 的 citycode (areacode.js)、東森房屋

rvest

A web scraper designed to work with magrittr.

• Create a html document with read_html()
• Select parts of a document using css selectors or xpath
  • html_nodes(doc, css = "<css selector>")
  • html_nodes(doc, xpath = "<css selector>")
• Extract components with
  • html_name(): the name of the tag
  • html_text(): all text inside the tag
  • html_attr(): contents of a single attribute
  • html_attrs(): all attributes
  • html_table: parse tables into data frames

Parsing Response Content: Text Type

Use with rvest

• 起手式

httr::GET(url) %>% httr::content(as="text") %>% rvest::read_html()

Beware of System Encoding

• Parse data with appropriate encoding
• Use Encoding(), iconv() and Sys.setlocale() in Windows (non-UTF8 system).
• iconvlist() shows all encodings availabe.

Encoding

?Encoding ?iconv

## check out your system locale
Sys.getlocale()
#> [1] "en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8"
aa <- "你好嗎"
Encoding(aa)
#> [1] "UTF-8"
charToRaw(aa)
#> [1] e4 bd a0 e5 a5 bd e5 97 8e

(aa_big5 <- iconv(aa, from = "UTF-8", to = "Big5"))
#> [1] "\xa7A\xa6n\xb6\xdc"
Encoding(aa_big5)
#> [1] "unknown"
harToRaw(aa_big5)
#> [1] a7 41 a6 6e b6 dc

# On Windows
Sys.getlocale()
aa <- "你好嗎"
Encoding(aa)
aa_utf8 <- iconv(aa, from = "Big5", to = "UTF-8")
Encoding(aa)

Extract elements from HTML document

Tree Structure of HTML (Document Object Model, DOM)

<a href = "www.meetup.com/Taiwan-R">
  Taiwan R User Group Website
</a>

• nodename : a
• attribute : href with value "www.meetup.com/Taiwan-R"
• text: Taiwan R User Group Website

A simple HTML document

demo_page

library(magrittr)
doc = readLines("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html") %>%
    paste(collapse = "\n")
cat(doc)

#> <!DOCTYPE HTML>
#> <html>
#>     <body>
#>         <div id='title' class='character'>
#>             <a class="link" href="http://data-sci.info/r-crawler-101/">data-sci.info</a>
#>         </div>
#>         <div id='summary' class='character'>
#>             <span class='title'>Number:</span>
#>             <span class='number'>2</span>
#>         </div>
#>         <div id='table1' class='info'>
#>             <table style="border-collapse: collapse; border: 1px solid black;">
#>                 <tr>
#>                     <th>Name</th>
#>                     <th>Gender</th>
#>                     <th>Age</th>
#>                 </tr>
#>                 <tr>
#>                     <td>Alice</td>
#>                     <td>Female</td>
#>                     <td>24</td>
#>                 </tr>
#>                 <tr>
#>                     <td>Jane</td>
#>                     <td>Female</td>
#>                     <td>26</td>
#>                 </tr>
#>             </table>
#>         </div>
#>     </body>
#> </html>

CSS practice
http://flukeout.github.io/

doc <- read_html("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html")
doc %>% 
    html_nodes(css = ".character") %>% 
    html_text

#> [1] "\n            data-sci.info\n        "         
#> [2] "\n            Number:\n            2\n        "

doc %>% 
    html_nodes(css = "#title > .link") %>% 
    html_text

#> [1] "data-sci.info"

Extract with xpath

doc <- read_html("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html")
doc %>% 
    html_nodes(xpath = "//*[@class='character']") %>% 
    html_text

#> [1] "\n            data-sci.info\n        "         
#> [2] "\n            Number:\n            2\n        "

doc %>% 
    html_nodes(xpath = "//div[@id='title']/a") %>% 
    html_text

#> [1] "data-sci.info"

Extract name of tag

node = doc %>% 
    html_nodes(css = "#summary") %>% 
    html_name
node

#> [1] "div"

Extract link

link = doc %>% 
    html_nodes(xpath = "/html/body/div[@id='title']/a") %>%
    html_attr("href")
link

#> [1] "http://data-sci.info/r-crawler-101/"

Example: PTT

use package xmlview to parse XML content and extract elements.

• PTT

Extract table

(rvest::html_table is temperarily not working with unicode data.)

students <- read_html("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html") %>% 
    html_nodes(xpath = "//table") %>%
    html_table()
students

#> [[1]]
#>    Name Gender Age
#> 1 Alice Female  24
#> 2  Jane Female  26

Extract table

use XML::readHTMLTable()

res_text <- GET("https://zh.wikipedia.org/wiki/%E5%9B%BD%E5%AE%B6%E4%BA%BA%E5%8F%A3%E5%88%97%E8%A1%A8") %>% 
  content("text", encoding = "UTF-8") %>% 
  `Encoding<-`("UTF-8")

# create `HTMLInternalDocument` for following use
doc <- XML::htmlParse(res_text, encoding = "UTF-8")
class(doc)

#> [1] "HTMLInternalDocument" "HTMLInternalDocument" "XMLInternalDocument" 
#> [4] "XMLAbstractDocument"

# Parse tables with XML::readHTMLTable()
tables <- XML::readHTMLTable(doc)

# str(tables) # take a look 
# View(tables[[2]]) # take a look 

• Result

Example: Yahoo Stock - (待改寫)

• Yahoo Stock

Example: 公開資訊觀測站

• 公開資訊觀測站

Parse XML table

Parse XML table with XML::xmlToDataFrame()

Example

• Seven Eleven
• RegEx: 7-11 citycode (areacode.js)

Parse JSON format

Yun can parse JSON with jsonlite in R

library(jsonlite)
library(magrittr)
res <- GET("http://ecshweb.pchome.com.tw/search/v3.3/all/results?q=sony&page=1&sort=rnk/dc")
res_df = content(res, as = "text") %>% 
    fromJSON() %>% 
    .$prods     # equivelent to (function(x) {x$prods})

#> No encoding supplied: defaulting to UTF-8.

str(res_df)

#> 'data.frame':    20 obs. of  11 variables:
#>  $ Id         : chr  "DYAI49-A9006HMLV" "DYAD2O-A9006JDQE" "DYAD2J-A9006I085" "DYAD2R-A9006XKSI" ...
#>  $ cateId     : chr  "DYAI49" "DYAD2O" "DYAD2J" "DYAD2R" ...
#>  $ picS       : chr  "/pic/v1/data/item/201509/D/Y/A/I/4/9/sDYAI49-A9006HMLV000_55f2883d1440e.jpg" "/pic/v1/data/item/201510/D/Y/A/D/2/O/sDYAD2O-A9006JDQE000_560e037b08db2.jpg" "/pic/v1/data/item/201509/D/Y/A/D/2/J/sDYAD2J-A9006I085000_55f7f34abd5c9.jpg" "/pic/v1/data/item/201604/D/Y/A/D/2/R/sDYAD2R-A9006XKSI000_56fe3f0b20c2e.jpg" ...
#>  $ picB       : chr  "/pic/v1/data/item/201605/D/Y/A/I/4/9/DYAI49-A9006HMLV000_57353014df35e.jpg" "/pic/v1/data/item/201604/D/Y/A/D/2/O/DYAD2O-A9006JDQE000_57198b2dc111b.jpg" "/pic/v1/data/item/201509/D/Y/A/D/2/J/DYAD2J-A9006I085000_55f7f34aba324.jpg" "/pic/v1/data/item/201605/D/Y/A/D/2/R/DYAD2R-A9006XKSI000_572ff4b5655b1.jpg" ...
#>  $ name       : chr  "SONY SmartBand 2 智慧心率監測手環_ SWR12" "SONY Xperia Z5" "Sony Xperia C5 Ultra E5553 6吋八核自拍機 " "SONY Xperia Z5 Premium" ...
#>  $ describe   : chr  "▼每日強檔‧瘋殺特賣▼SONY SmartBand 2 智慧心率監測手環_ SWR12" "超值▼送32G卡+保護套+保護貼▼SONY Xperia Z5 5.2吋美型防水旗艦機" "送保護套+玻璃保貼Sony Xperia C5 Ultra E5553 6吋八核自拍機 " "▼送32G卡+手機立架+玻璃貼SONY Xperia Z5 Premium" ...
#>  $ price      : int  3180 16900 8890 19900 11900 16777 20900 4680 2240 13490 ...
#>  $ author     : chr  "" "" "" "" ...
#>  $ brand      : chr  "" "" "" "" ...
#>  $ publishDate: chr  "" "" "" "" ...
#>  $ isPick     : int  0 0 0 0 0 0 0 0 0 0 ...

res_list = content(res, as = "parsed")
str(res_list$prods[[1]])

Parse json with loop

res_df2 = data.frame()
for (i in 1:length(res_list$prods)) {
  res_df2 = rbind(res_df2, 
                  data.frame(res_list$prods[[i]], 
                             stringsAsFactors = FALSE))
}
identical(res_df, res_df2)

Useful R functions to combine list of dataframes

• do.call + rbind
• data.table::rbindlist

Parse Parse list of data.frames with do.call + rbind

res_df3 = data.frame(do.call(rbind, res_list$prods))
identical(res_df, res_df3)

# An ugly data.frame
str(res_df3)

非結構化資料解析: Regular Expression

Package: stringr - str_replace(), str_replace_all() - str_extract(), str_extract_all()

library(stringr)
fruits <- c("one apple", "two pears", "three bananas")
str_replace(fruits, "[aeiou]", "-")
#> [1] "-ne apple"     "tw- pears"     "thr-e bananas"
str_replace_all(fruits, "[aeiou]", "-")
#> [1] "-n- -ppl-"     "tw- p--rs"     "thr-- b-n-n-s"

shopping_list <- c("apples x4", "bag of flour", "bag of sugar", "milk x2")
str_extract(shopping_list, "[a-z]+")
#> [1] "apples" "bag"    "bag"    "milk"
str_extract_all(shopping_list, "[a-z]+")
#> [[1]]
#> [1] "apples" "x"     
#> 
#> [[2]]
#> [1] "bag"   "of"    "flour"
#> 
#> [[3]]
#> [1] "bag"   "of"    "sugar"
#> 
#> [[4]]
#> [1] "milk" "x"

Exercise : 東森房屋

• 東森房屋
• 試著找到「新北市」的資料

Exercise: 紫外線即時監測資料

行政院環境保護署環境資源資料開放平台提供了一系列的RESTful Api供大家取用，請試著把 紫外線即時監測資料的資料取回來並轉成data.frame

Answer

紫外線即時監測資料

Save your data

• write.csv: write data.frame as csv file
• download.file: save html, jpeg, etc
• writeBin: write binary object into disk
• RSQLite: SQLite connector in R

write.csv

library(jsonlite)
library(httr)
url = "http://ecshweb.pchome.com.tw/search/v3.3/all/results?q=sony&page=1&sort=rnk/dc"
res_df = GET(url) %>% 
    content(res, as = "text") %>% 
    fromJSON() %>% 
    .$prods     # equivelent to (function(x) {x$prods})
write.csv(res_df, "resources/data/pchome.csv", row.names = FALSE)

download.file

To download a file from the Internet. download.file takes advantage of internet utilities such as curl or wget and may fail if you don’t have any of these utilities in your system.

dest_dir = "resources/data/download"
dir.create(dest_dir, showWarnings = FALSE, recursive = TRUE)

# Download whole HTML file
download.file("https://www.r-project.org/", 
              file.path(dest_dir, "r-project.org.html"))

# Download image
download.file("https://www.r-project.org/Rlogo.png",
              file.path(dest_dir, "Rlogo.png"))

list.files(dest_dir)

writeBin

To write binary data to your local disk.

r = GET("http://opendata.epa.gov.tw/webapi/api/rest/datastore/355000000I-000004/?format=json")
# Set as = "raw" to prevent any character encoding
bin = content(r, as = "raw")
writeBin(bin, "resources/data/download/uv.json")

Database

• RSQLite: A light-weight database engine interface in R
  • PChome Example
• Other relational database connector in R
  • RODBC
  • RJDBC
  • RMySQL
  • …

Case study

• 東森房屋
• Guestbook
• Yahoo Stock
• 紫外線即時監測資料
• PTT Gossiping
• Starbucks
• Seven Eleven
• FamilyMart # jsonp 待補
• PChome
• 公開資訊觀測站
• 中央氣象局觀測資料查詢系統

References

• rvest
• httr
• HTTP Methods
• An Introduction to APIs
• Advanced R
• data.table
• RSQLite